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A^BSTRACT . , , 

Rating scales of the ^scaled behavioral €ipectation^ 
type vere developed to measure the constructs ot independence and 
'initial ivd# conscientiousness* eilthusiasi* critical facility # 
teaching skills, research and experimentation, ccanunication, and 
p^sistence* The scales vere used by iaculty in three psychclogy 
departments* tvo chemistry departient£# and cne^English department # 
vho indicated the^skills and behavior vhich vere expected ot graduate 
students in their departkent* The following data v^re al£C collected: 
faculty fie&ber*s estimate of confidence fcr each rat'ing; Graduate 
Record Examinations ^fGRE) Aptitude and Advanced lest sccxes; amcunt 
of graduate .vorjc cos^pleted; and undergraduate aod graduate grade ' 
point average* The scales vere found tc have cnly jinimal reliability 
and rathe^ high intercorrelations. Further research* cn the scales is 
necessary before they can be used vith ^y confidence* the available 
data did not suggest any striking relationship betveeAG££ scores and 
faculty ratings; thus* it is doubtful that such rat ijg ^c^ulA be 
useful as criteria in GGE validity studies* (Author/9^) 
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Abstract 

Rating scales of the "scalsf'd behavioral expectation" type were devel- 
oped to measure the constructs of Independence and initiative, conscientious^^ 
ne8s» enthttslasm^ critical facility* teaching skills^ research and exper-* ^ 
imentatlon^ communication, and persistence, The scales were used by faculty 

In three psychology departments, twO' chemistry departments, ^nd one English 

\ - ' 

departinent. 

The rfcdles were found to have only nilnlinal reliability and rather high 
intercorrelatlons. Further research on the sqales' is "necessary befor^ they 
can be used with any confidence. ^ ' ^ , ^ 

4 ■ * ^ 
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The Development and Pilot Testing of ^ 
Criterion Rating "Scales 



Obtaining useful measurements of the performance, of graduate students 
has been. a persistent concern for researchers In assessing .the validity of 
tests ^r other Instruments In the graduate school context, Lannholm, Marco, 
and Schrader (1968) discuss the difficulties In assessing the predlctlife 
validity of admissions Instruments. Rellly (1971) sinomarlzlng some of ^ 
the^e difficulties stated^ > - 

First, the small sasz{)l^ sizes available at the graduate , 
level make results, especla4;ly when several predictors are In- 
volved^ 'subject to a considerable degree of error. Second, the 
fact that students Within a g>ven dMggtment have^ gone through 
an elaborate screening process end^^^&^^^ult ^e iisually 
quite homogeneous with respect to^l^^^^^fietpAnformatlpn, 
often leads to restricted varlatlo^^^^^Rredlctor scote dis- 
tributions* Finally, there Is the TBrSHIty^of establishing 
an adequate, crldejrlon of graduate school^^erformance; Grade 
point average (G?A), while It has been. the qiQst widely, used 
crlt^lon, has also been the most severely crit.i^iz^tJi^ Far- 
haps the most Important and valid of th^se ^itlcisms:4s that 
the 6FA represents only a limited aspebt of *gtadua'te school 
performance t (p, 1) . . * ^ * ' ' 

In a later report .to the GRE Board Research Committee, RelHy (1974) 
summarized the results of a two-^ph^^ study that was aimed at empirically 
defining dimensions of gradt;iat& student performance. During the flr^t 
phase of the study,, a serlea of ''critical'* incidents which reflected un^ ^ 
ust^ally effective or unusually Ineffective performance was collected from 

faciulty members. A final edited list of 52 indd^octs was then used as a 

i 

checklist by the faculty in departments of chemistry, English, and psy- 
chology to evaluate the performance of selected students. 

The study resulted in the identification, through factor analysis, of' 
elgj^t relatively coherent, dimensions of performance represented by clusters 
of incidents, Ihe eight dimensions were independence and initiative, con- 
scientiousness, enthusiasit, critical facility," teaching skills, research 
and experiment at iotr, commtjnicatlon, and persistence, A fair degree of 
consistency existed across th!e three fl^elds^ and the identificajtion of 



these fsctors served as s first step In the development of s tissble set 
of crlterls for assessing graduste student performance* , 

> The pui^^ose of the study reported here was to develop s set of %e- 
haviorally anchored" rsting scsles by which the factors identified in the 
first phase'^of the study could be measured and to pilot test the scsle^ The 
pilot study was to explore the feasibility of grsduate ^fsculty using the' 
developed scsles and to determine the psychometric sdequacy of the scales. 
It would slso serve as a wsy of obtsining fsculty rescfion to specific 
aspects of the scsles. 

Rating scsles hsve been the subje<^ of^resesrch by psychologists for 
many yesrs. Although there ^re many problems with rating scsles (Guilford^ 
1954) » their appeal to evaluators and resesrchers alike is sufficiently 
great that they continue to be one of the most commonly used techniques 
of evslustion* The **scsled behsvioral Cxpectstion" technique developed 
by Smith and Kendsll (1963) is generafly recognized^ss s technique thst 
can effectively overi^ome. many of the problems thst seem to be^ inherent in 
using rating scales. This technique is designed to provide as much help 
to the rster as possible In making his judgments. Expected behsvlors. sre 
used to Encourage him to be conscientious; involvement of the rsters'^eers 
is 'intended to maximize communicstion througl^use oft^ ^pprojpriete terminology 
and to injure s^high degree of content validity. 

However > sets of rsting scales thst hsVe been developed usin^ the'^scaled 



behavioral expectstion technique tend to reqy.re that th,e rster hsve s very 
thorough scquaintance with and knowledge of the retee. In the context of 
grsduste business school^ it was found j^st m£|^ rsters (professors) did 
not know the rstees (students) well enough to mke informed rstings on many ^ 
of the dlD^jensions (Hilton, Kendsll, A-Sprecher JJa970), .For this resson^ one 
of the main purposes of the pilot study wss Ito ¥Bplore the {essibility of ^ 
using the scsles, Not only wa& there a^ qtie8tioiQt|f whether the fsculty^ 
would know the students well enough to make sde^iiK ratings, but there 
was slso the question sbout the^^llingness of 4^B^^^°ts and fsculty to 
utilize the scsles for purposes of resesrch, ^ ' 



A stuHy'of common criteria Iritgrsduate education by Carlson, Evans, 
and Xuykendall \l913} suggested that rating scales would be ^ acceptable 
criterion measure In many fields. Many departments use rating proceduras 
to selept students for financial aid, to determine which students will be 
encouraged tfr continue, etc. In such cases, these ratings- serve a pax- 
tlcular nee<L for the department', and t)(^ procedures are developed for a 
specific purpose* ^'In addl^on t*9 being perhaps more difficult to complete, 
a more general set of scal^ for evaluating graduate students may not be as 
useful to the department* Also, many faculty members sre geiiuinely con- 
cerned about -Confidentiality of evaluative ihfofmatlon , and others are con- 
cerned because qf recent laws and rulltigs. Thus,tthe question of the willing 
ness of faculty and departments to utl^ze the scales, particularly for re- 
search purposes, is a very real one, 

Another purpose of the pl^ot study was to examine the scales from a 
psychometric poiilt of view — the interrater reliability of each scale, th^ 
correlations among the scales, and the corr^S^tlons between each of the 
scales and other Information available on the studenta* The Interrater 
reliability. of a scale can be thought of as the avarage correlation between 
raters when they are rating the aame group of students^ Ideally, each scale 
would have a reliability of at lea&t ,S0» although many rating scales that 
have beeif developed do not achieve this level. If this level can be achieved 
however, it is poasible to obtain a reasonably reliable rating of an indi- 
vidual by averaging the ratings assigned by BeveraT raters . 

At the same time, the correlations among the acales wpuld Ideally be 

les^ than ^.50i The construct that each scale is developed to measure was 

1 ' * 

identified from an analysis of Independent factors; if the correlation among 
the scalas is high, then the scale is not, adequately measuring the factor con 
struct* Also, if the correlation between two scales is relatively high, then 
only one of the two scales is necessary alnce they are maasurlng essentially 
the same construct. 

Finally, if at leaat some of the scales are aufflclently reliable and 
hava only small or moderate corralations wi;:h each othe?:^ the corralatlon of 
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these BCal^s-^ith other Information on the student should be. consistent with 
logical expectations concerning the meting of the scale In question. For 
example* one would e^ect that thd correlation between GR£ scores and ratings 
of "critical facility" would be higher than the correlation between these 
scores and ^V^i^sls^^^^** ratings. At the same time, imdergraduate grades 
may well have a moderate or even high correlation with "psi^sistence,'* 

The e^casdnation of correlations was a secondary p^urpose of the pilot 
study. Participating departments were asked to supply readily available 
"predictor** data\(such as GRE scores or imdergraduJPe grade-point average) 
^and ''criterion" data (such as*routinely obtained departmental ratings, ^ 
gradiiate grade-point average, or prelim scores), ^ese data could pro- 
vide useful information for Increasing the understanding of the '^meaning" 
of the scales. 



Method 



/•■■ 



Scale Ifevelopment ,^^/^ 

^pr^^li taken In developing the scales was conalstent with methods 
^alth and Kendall (1963) in the description of the scaled be*- 
ei^ectation technique. Briefly, the steps were as follows; 

1, A gener^ definition of each scale was written, 

A pool of specific behavioral examples for later aaslgnment to the 
cal^s was prep'ared and edited to appear in a common format. The original 
list of 52 Incidents was part of this pool, as were other Incidents suggested 
hy faculty respondents dij^rlng the course of the previous study. Behavioral 
examples were also culled from those collected for the ATGS^ Criterion Study 
(Hilton, ICendall, ^ Sprecher, 1970), Ii^ addition, a number of new examples 
were written, many for the mid"-ranges of the scales, aince the orlginal'llst 
of behavioural examples represented only extremes of performance. 



3, A pool of 12 judges (four froair^each dleclpline) assigned behavioral 
examples to scalea based on their judged relevance to the general definitions 
in Step 1. Particular attention was paid to dlsagreament between fields on 
any of the examples. Only examples for which there was strong agreement 
aoong judges were retained. 

A. Scale values were assigned to the subset of behavioral examples 

assigned to each scale. The same pool of judges rated each example In terms 

of the degree to which they thought that it reflected effective (or ineffective) 

performance on the continuum defined In Step 1, The distribution of scale 

values and other Information for each behavioral example are given In Appendix A* 

' T 
5a a final set of behavioral examples selected to anchor each scale was 

based on two criteria. First, the decree of agreement a^ng judges was con^ 
slderedt Exa m ples where there was lack of agreement ^n scaled values were 
eliminated. Second, a set of exaiaples which covered the entire scale con- 
tinuum was chosen so that only one of two or more examples with the same, or 
nearly the same, scale values were retal-ned. For these scales, from five to 
seven examples were considered sufficient to represent an^ors over the range 
of scale values. The final set of rating scales Is Included In ^pendix B, 
pp. 47-56. , ' ' ^ 

Data Collection , 

Plans for the data collection included obtaining cooperation Irtm at 
least three departments within eacli of the disciplines studied; eati^ depart- 
stent should be large enough to anticipate providing ratinga for at' least 75 
students. An effort was niade to restrict student ratees to thos^ who had 
completed att li^ast two years of graduate study and to .have at least* two- ^ 
thirds of the students rated by two faculty members i Within each depart- 
ment, a coordinator waa designated and given a set of detailed Instructions, 

^ • ' 

(see Appendix B) , " 

In addition to ratings by faculty, the following data were collected 
where possible: ^ ^ v " , 

0 ' . 



li Estimates of the confidence a faculty membet had for 
each rating made; , 

' ^ 2* GRE Aptitude TesJ^cores; 

3* SRE Advanced Test srores; 

4, Number of-semesters (or qiiarters) of gradate ^tudy.; 

5. Undergraduate 6PA; 

- *6, Graduate 6PA, and 

7, Any other measures of graduate student performance 

routinely available (eig.* departmental ratings, class 
rank, Ph.D. prelim scores). 

I 

Analyses , ^ 

Means, standard deviations , aii^dfiore ranges were computed for each 
rating scale to provide Informatloll on Jthe extent to which faculty used the 
full range of the scales^ Frequencies of confidence estimates were also 
tabulated^ 

Soale reliabilities were estimated for the'subsample o£ students with 
two ratings for each scale throu^jh an analysis of variance procedure described 
by Winer (1962, p. 126). Reliability of the average of two (or more) raters 
Is estimated ae 

\ • ^ MEAH SQUARE WITHIN RATSES 

"'jfflAN SQUAitE BETWEEN RATEES * 

Jhe reliability for one rater Was estimated by the Spearman -Brown fonnula 
(Guilford, 1954, p. 354). ^ ) 

The conflations ai&ong th^ scales and correlations between scales and 
other 4ata wer^ computed for each department. (The "secoh'd*' r&tlng for each 
student who had two ratings was not Included in this analysis.) Since most , 
of these sample sizes were quite small, a significance test was computed for 
each of these correlations. The significance level chosen was «0l because of 
the large number of coefficients being considered. Data were pooled within 
dlacipline to e:camine the correlations among the scales. Because the other 
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data collected were so 'Scattered , It was decided to compute correlations 

i 

with these variables on a departmental basis and to present these corre-^ 
latlons only for ttfbse variables for which sample sizes were minimally 
adequate. 

* Departmental and Faculty Cooper^lon 

. / ■ 

Initial contacts with departments were generally made. through the 
graduate ^ean* 8 office. The purpose of the project was explained, and the 
most expeditious manner of sot'iclting the cooperation of the appropriate 
departments was discussed. .With the dean's approval, department chal^en 
w^e then approached. As mentioned previously, original plans called for 
participation of approximately nine departments, thr&e each in the fields, 
of chemistry, psychology, and English. Seven institutions were, contacted. 
The final distribution ^f participants in the study is shown in Table 1. 

Although cooperation* was obtained from department chair^n, there was,^ 
of course, no guarantee that faculty would agree to participate. Procedures, 
for gathering data were purposely informal with the hope that this would maxi- - 
mize the amount of information collected from the various departments. How- 
ever, considerable faculty resistance was encountered, and^ the *^amples ob- 
tained were s&aller than anticipated. Suggested reasons for tq| high rate 
of refusals are multiple, complex, and varied among' departments^ therefore, 
no attempt has been, made to order them by frequency or important* The ^ ^ 
reasons appear to include: (a) faculty time pressures, (b) a ge&eral dis-* 
like of tatirtg scales, (c) lack of familiai^ity with a student's ^ork, (d) the 
feeling that ^hese rating scales did not pertain to the kind of evaluation of 
students that took place in the department, and (e) the general tea^r of the 
times /concern about the Buckley Amendment; resistance tro any sugg^tion^ of 
invasion of privacy or contribution to data banks In the ^ra o£ gtfJArtng in-^ 
formation deposits). ^ \[ ^ 

* Table 2 sumarizes the amount of data actually provided^by gra|[\iate 

. - - ^ . 

departments. Psychology ^as. the "only area in which reasona^^e si 

r 
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Tflble I 

OlsCribuClon of Participants in the Study, by Oepirtmint" 



Department 


Hufi^«r of 
D«partmttnts 

Contacted 


Numiber Agreeing 
to Participate 


HuBiber Frovildlng 
data 


Psychology 


A 




3 


ChjY<^stry 


A 


2 ; 


2 


Eiiglis^ 


6 , 


2 


1 



Table 2 



Distribution by,Departneiit of^Ratings and Additional Information Obtaineji 



Department 



C 
4) 

5 



s 



O 
4) 



'H 4) 



i2 



CO 



4) 

§ 

> 



o 

M 
4) 



J 



O CO 
41 

Ti M 

D m 

o 9 

:^ 

0 4) 

o m 



w 
n :^ 










M U 
CO 




1-4 




1 


O M 




'1 i 


-P 


55" to 





4) 
CO 



CO 
M 

to 

M 
4) 

I 



Psychology 



C 



Total 



Chemistry 



77 
43 
36' 

156 



A 



Total 
EnRllsh 



10 

111 

23 



55 
25 
25 



105 

26 
lA 



AO 



■13 



13 
33^ 

34 



r3 

33 
3A 



10 
20^ 

30 



A6- 



(75) 
'( 6T 



7S 
39 
3A 



80 



80 



60 



A6 



81 



148 



lA 
12 



lA 
lA 

12 



13 
13 

11 P 



35 
35 

- — * — 



A3 
A3 

23 



76 
A3 
36 



155 



A3 
A3 

23 



A5 
2A 
36 



105 



A.A 
' AA 

22 



er|c 



' were obtgined^for th^ purpos^ of a reliability analysis* Data^^n other 
Variables were genetally sketchy, particularly for GKE scores* The nbst 
consistently provided additional data were graduate and undergraduate^ GPA* 
Overall* the totals for psychology and chemistry* though smaller than 

— ^dTtlctpated, could be considered at least minimally' acceptable for «the 
pilot study* For English^ however, th6 data provided are clearly of very 
limited value* especially for p'uVposes of estimating rating 'Scale^rella" 

_ blllty* 



Results of Data Analyses 

A first ccAtcera with respect to the usefulness of any set of rating 
scales id the extent to which^the entire range of the scale 18 utilized by ^ 
raters* Nopoally one would expect raters to be on the lenient side with 
ratings but to make at least iseveral ratings at' the lower end c^^the scale* 
Tables 3> 4» ^and 5 present means » standard deviations » an6 score ranges for 
each discipline studied* It is clear from these tables that raters do tend 
to use primarily the higher end of the scales*" However, it is tfjso cle'ar 
from the.^high and low s^cores giv^ within each departitent th^t most of the 
range is being utilized* The average rating given in most departments was 
'near 4*0, tod the st^dard deviations £o^ each scale averaged approximately 
7/1(1 of a scale unit* 

* 

Because the rating scales represented a common set of Variables, it was 
decided to pool rating scale data across departments within discipline so* t^t 
more stable ^estimates of soale intercorrelations and^ reliability ^could be de*t 
rivedt Table 6 presents scale intercorrelations and reliabilities for one and 
two raters for psychology*^ With the eXQeption of Critical Facility, the scales 
ss used in psychology departments appear to possess at least modest reliability 
^ f or two ra£ers* Correlations between scales, however, are high considering 
the level of Reliability^ si^ggesting a marked halo effect. Table 7 presents/ 



Intercorrelations for Tables 6 through 11 were based on the maximuct 
amount of data available for each computation. Ttnus, the numbers of cases 
actually used vai:y slightly from thos^ reported in Table 2* 



Table 3 . 

Meansi Standard Deviations and Score-ilanges " 
for Rating Scales in Thre& Psyjchology DejJartments 



Psychology A 



Scale \ 


* 






Scotre Saa£« 


GoiftKunlcatlen ^ 




3,9? 


0.74 


5.00 


* jr. 50 


Cons|cilentlousness 




4.12 


0.85 


5.00 


■ 2.00 , 


Critical Ficllty 




4.16 


0.66 


5.{)?> 


■1.50 


Independence and Initiative 


4.10 


0.82 ■ 


5.00 


1.75 


Involvement 




4.11 


0.76 


5.00 


. 1,50 


Persistence 




4.08 * 


0.70 . 


<5.oo 


1.75 


Research 




^.02 


0.77 


5.00 


1:25 


Teaching 




4.07 


■ 0.67 


5.00 


1.00 



Scale 



Paychology B 
H&an 



S.D. 



Score Range 



Coiamunication 

Conscientiousoess 

Critical Facility 

Independence and Initiative 

Involvement , 

Persistence 

Research 

Teaching ^ 



3.65 
3.89 
*3.80 
3.88' 
3.84 
3.93 
3.62 
4.05 



J 



0.70 
0^81 
0.75 
0.83 
0.75 
0.70 
0.89 
0.56 



4.75 
5.00 
5.00 
5.00 
5.00'" 
5.00 
'4.75 
75.00 . 



1.50 
2.«0 

It^ 
1.25 
3.00 



4, 

Scale 


PaycholoRy C 
Hefan 


S.D. 


Score Raase" 
Hiflit '-Low 


Communication 


3.92 


0.57 


5.00 


1.75 


Conscientiousness 


4.11 


0.87 


5.0Q ■ 


1.50 


Critical. Facility 


3.91 


^0.92 


5.00 


. 1.00 


Independence anid Initiative 


3.87 - 


.0.93 


5.00 


i.:>5. 


Involvement 


3.85 


0.88 


5.00 


. 1.25 


Persistence 


4.03 


0.69 . 


5.00 


2.25 


Research 


3.83 


0-.81 


5.00 . 


i.po 


Teaching 


'3.96 


0,84 


5.00 


1.50 
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Table % 



Meaqa, Standard Deviations, and Score Ranges for, Rating 
Scales in Two Chemistry Departments 



ScAle 



Chemistry A 
Mean 



S.D. 



ScOM Rgngft 



CoiBDtunlca%lon " 
Conscientiousness 
Critical Facility 
Independence and initiative 
Involvement 
Persistence 



R&search 
Teaching 



3.62 
3:91 
'3.93 
3.63 
3.66 
3.99 
3.81 
3.88 



0.68 
0 .80^- 
0.53 
0.97 
0.78 
0.72 
0.64 , 
0.73 



4.50 
5.00 
5.00 
5.00 
5.00 
5.00 
'5.00 
5.00 



1.75 
^.00 
2.25 
1.25 
2.25 
1.75 

1.56 



ScAle 



Chemistry B 
Mean 



Contminlcat ion V 

ConScientiousn^B 

Critical Facility 

Itidependence and Initiatiife^ 

Involvement 

Persiatence 

Research 

Teaching 



3.86 
4.07 
4.12 
4.18 
4.10 
4.09 
4.p2 
4.08 



S.D. 



0.77 
0.9j6 
0.62 
0.69 
0.71 
0.74 
0,48 
0.84 



Score Range 
Hligi Low 



4,75 

5.eo 

5.00 
5.00 
5.00 
5.00 
5.00 
5.00 



1: 

1.50 
1.75, 
1-25 
1.75 
3.00 



.cm ■ 



( 



. ^ Tat)le 5 \ * 

Iteans^ Standard Devlatlond».and Score Ranges for 
Rating Stsiailes in One English Department 





aDglZSu 








Scu« 


Mean' 


> 

S.D. 


Scor« Raiui« 
Blfib Low . 


• 

Comunlc^tion 


4.07 


0.87 


i 4*75 


'^55 


Conscietltlousneas 




0.^9 


5.00 


' 2.00 


Critical Facility 


4.31 


' b/81 


5.00 


. 1.47 


Independence and Initiative 


* A.. 25 


'{;l'.63 


5.00 


2.13 


Involvement 




'0.75 


4.75 


1.63 


Persistence 


4.25' 


0.75 


^0 




Res earch 


4.22 


' . 0.79 




1.59 


Teaching 


4.19 


0.91 


3.00 


. 1.74 






Hating Scale ,Intercorre\atloQ3 and Reliabilities 
Pooled Over Three Psy^ology Departments 



Scale 



CoOSDUtll** 

cation 



Conscien- 
tiousness 



Critical 
Facility 



Indep . and 
In^latlve 



Involve- 
ment 



Persis- 
tence 



Communication 

Conscientiousness 

Critical Facility # 

Independence and^ 
Initiative 

Involvement-^ 

Persistence 

Research - 

Teaching \ 



.45 



.49 
.'44 



.44' 

,55 

.48 



.49 
.54 
.43 

.64 



.32 
.46 
.35- 

.62 
.56 



.47 
.62 
.48* 

.78- 

.58 

.61 



Reliabilities^ 
9 For one rater .25 
For two raters ■.'40 



.47 
.64 



.10 

.19 



.25 
.4ff 



{ ,33 
.50 



.36 
.53 



.30 
.46 V 



Research Teaching 




.25 
.40 



17 



Table 7 - 
Rating Scale Inter correlations and Raliabllltles 



■\ ■ ■ \ 




I. 




Fooled Over Tvo Chemistry. Separtmenta 


* > 


1 

1 

i 




I i 
* 

S<iale 






Comininil'^ 

cation . 

if 


Consc^en^ 
tlous^eas 


Critical 
Facility 


. Indep . and \ 
Initiative 


Involve-- 
laent ' - 


Persis- 
tence ' 


Research Te 


^chlng 


Xonnzrunl cation 






r 




.55 


.40 


' .51 


■A T 

. .39 


" .61 


1 — 
.■54 




Cons clen tlousnep 


s 






.45 


.44 


.47 




,65 


^3 




Critical Fadllt 


y 








.42 


•'.59 


.39 


.55, 


.36 




Indep^endence an< 
Initiative 












.48" 


.49 


.71 


.28 




Involvement 
















.56 . 


.58 


.38 




Persia tence 


















.62 


.29 




Research 




















.42 

t 




. Teaching 




1 










■J 






1 




Reliabilities 




! 


7-= 

/ 
















t 


, For one rater 


1 

1 




.38 


'.37 


.24 


'.17 


.00 


.27 


.43 




For two raters 


i 


,67 


.55 

♦ 


. .54 


.39 

L 


.29' 


.00 


.42 


.60 


1 . 






1 


















1 ' 




. . • . -15- 
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rellabffftte8 and Intercorrelatlons for chemistry departments, six of the 
e<^ee have at least inodest reliability for two rater«. The persistence ^ 
scale was completely unreliable as estimated from these data> and the re- 
^ ^ liability for the involvement ^cale was very low (*17). Again, the inter*' 

correlations aniong scales are high considering the level of reliability. 

Table 8 (based on a very small sample, as noted previously) presents 
correlations between scales and reliabilities tor the one English depart** | 
ment that provided data. Allowing for rater sampling variance, the pattern 
Is consistent with that observed in the other two dlsoipllnea. That Is^ ■ - 
7 % vlth one or two exceptions Scales have at least moderate reliabilities for 
' two raters and scale Intercorrelatlons are high. 

Table 9 presents correlations between rating scales and selected varl" 
^ables f6r eac^ of the three psychology departments* The largest number of 
correlations significantly greater than zero are In the table for Psychology 
department A. All of the correlations between average prelim scores and 
rating scales were significant* Aside from that set oJL^cprrelatlons^ how- 
ever, few of the remaining relationships were significant. 

Only one chemistry department was able to, provide ^iddltlonal data. The" 
"; results » presented In Table 10, are unimpressive. The only significant re- 
lationship out of 48 computed Is a negative" correlation between rank-ln-class 
^ and ratings on the teaching scale. In the English department (see Table 11)^ 

^significant correlations were obtained between three rating scales ,and gradu- 
ate grade-polnt average, but none of the other correlations reached signifl-^ 
cance. . ^ 

A flnal^set of data is presented in Table 12. The .percentages of raters 
expressing various levels pf confidence for each^ rating. suggest that an over^ 
whelming majorlty^of raters felt at least ''fairly confl^nt** In their ratings. 

- * * Discussion and Conclusions 

In the authors^ Judgment, the development of the rating scale? went very 
well, and all of the usual criteria for suoces^sful development of scales of 
this sort were clearly met, , ^ . ' 
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Table- 8 



^tlng Scale Interoorrelatlons aad Reliabilities 
' In One aigllsh Department 



Scale 



■4 



Comaunl- Consclen- Citltlcal tndep. and Involve- Fersls* 
cation tlousness Facility Initiative Btent tence 



Besearch Teaching 



CcRsnttinl cation 

Consclentlt)U8ne88^v^ 

Criiical Facility * 

Independence and 
Initiative 

Involvement ' 

Persistence 

Research 

Teaching 



.55 



.66' 
.6-2 



.68 

.64. 

.46 



.53 
.53 
.06 

.66 



-.22 
.50 
.27 

.02 



.93 

.70 

.62 
-.'38 
-.26 



.87 
.39 
.56 

.58 
.45 
-.22 
.81 



Reliabilities 
For one rater 
For- two ratets 



,34 
.51 



.00 
.00 



.16 
.28 



.61 
.76 



.24 
.39 



.34 
.51 



.26 
.41 



.42 
.59 



Table 9 



Correlations Between Rating Scales and SeXect€£d Vai;lables 
•in Three Psychology Departments ■ 



Psychology A 



Scale 


Dnd ere r a d ua t e 

GPA 

^ 


Graduate 
/ GPA 


GRE 
Verbal 


GEE 
Qoaiit 1 ta 1 1 ve 


GRE 
Advanced 


Average " • 
Prelint Score 


Miller 
Analo^es 


Cbmnunlcatlon 


-01 


•10* 


-.22 




.'57 




.09 


Conscientiousness 


.43 


.36 


-.13 


-.02 


. -.28 


.32* 


,;o7 


Critical Paclilty 


.31 


.09 ■ 


-.19 


-.03 


.10 


.23.. 


■ .07 


Lnaepennence d iniciative 


^ • i / . 




.03 ■ 


■ .01 


-.Oft 


i* 


.16 


Involvement 


.13 


.26* 


.00 


-.29 


.15 


.20; 


.10 


Persistence 


^ .17 


.20* 


-.06 . 


-.24 


^.36 


.33j^ 


.01 


Research 


.25 


.28 


-.09 


.08 


-.02 


•36* 


.11 


Teaching . 


.10 


.08 


-.14 


.-.19 


.11 


.42 


.oS 






Psychology & 










Scale I ^ 


Undergraduate 


Graduate 


GRE 


. GR£ 


,GRE 








GPA 


GPA ' 


Verbal 


Quantitative 


Advanced 




i 



Coomunickcioy • 

Conscientloi/sness . 44 

Critical t^cility .19 

Independence & Iniciacive -.05 

Involvement .14^ 

Persistence .48 

Sesearch ' 1 6 

Teachlne. ' .22 



.12 
.18, 

' .40 
.27 
.2$. 

-.01 
.12 
.30 



.26 
.07 

r.04 
.04 
.22 
.10 

-.00 
.12 



-.03 
-.28 
-.11 
-.03 
.07 
.13 
-.U 



.03 
-.35 

.04 
-.10 

.11 
-.-03 
.-.00 
-.18 











• 1 r ' 










Psychology C * . 








Scale 


Undergraduate 


Graduate . 


GBE 


GRE 


GRE . . - 




* 


■ GPA 


'GPA 


Verbal 


Quantitative" 


Advanced 




Communication 


.13 


i 

.42 


.22 


.'05 


.31 




Conscientiousness 


.23 


•.3X 


-.03 


^-.01 


.11 




Critical Facility 


.19 . 


.21 


-.0^ 


.04 


-.00. 




Independence & Initiative 


.23 


. .30 m 


.31 


..26 - 


.45 




Involvement 


.04 




.02 


.24 


.21 




Persistence 


.08 




r24 


.16 , ■ 


.20 




Sesearch 


,35 


.34 


.25 


.03 


.33 




Teaching , 


'-.06 


-.21 


-.17 


.28 • 


.02 





Significant at .01 level. 
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' CorrelatiLoQS ]^tveen Bating Scales and Selected Varlftbles 










" - in One Chemistry Departiaent 








i 














^ Scale 


Ondergr^uate^ 


Graduate^ '&RE - GS£ 

t&A Verbal QuantltatlvaT ^ 


GEE ^ < 
Advanced 


Rankr^ln- 


Ctnnmunicatlon 




■ L 

-.04 ' 


.23^ ' ^ .A3 ' 


. -.24 ^ 


.23 . 


-.0: 


jpoosclent^usneds 




.Ifr 


.26 ■ v51^ 


-.24\ 




-.01 


^Sfltical Facility 






.33 .32/ 


.04 


.33 


• . .o: 


Independence &,Inltiatlye 


' -.17 




t 


■.lf« 


1 


{Involvement * 






.21 ' -;12 


.01 


-.42 


-.0 


Persistence / 
;Itesearch ' > 






-.03. -.is- 


• -.02 • , 


-.54 






-.28 


.08 .01 


-.03 


'^0 


■ -.0 


Teaching 




-.01 


-.qi, :.27 * 




. -.10 ' 




:*Sla^*flcant at .01 level. 


- 


t * 


- ' . . 










- 




f 




















F 










V 








• 


f 






« I 














• 






1. 




f 


• * ' 



Table tl. 



-Correlatioiis Betveea Eating Scales .and Sel^ted Varia1)led 
in One l&gllsh Department 



: * Scale 


Ihidergraduate 
GPA* 


Graduate 
GPA 


GRE 
Verbal 


GRE 

Ouantitrative 


GRE [ Has 
Advanced' C 




• 


55* 
.16 


in 

- 11 


ir 

11 


- At 

.25 


^Critical Facility 


• .02 ' 


' ■'.16 


'-• ■ ..32 


46 


independeo^ "4-* Initiative 


.10 


_* 
.55 


.22 


.28^ 


.32 


llnvolvementl 


-.04 . ' 


.36 ■. 


-.34 


-.0^'- 


.32 . -. 


pfersisfience 
1 Re'^search 


-.40 


--.38 


-.26. 


-.28 ' 


-.39 00-- 


.27 • 


.58 


.".lis 










.52 


.17 


.16 


.45 


■ 












|^^ifi!cant at .bll^evel. 
1 A 




r 

/ 









Rir 
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Table 12 



Percentages of Faculty Xndlcatlag Various Levels 
of Confidence In Ratings 





Psychology 




Chemistry 




! — ' ; 


Etfellsh 




^ 


Very 
^Confident 


Fairly 
Confident 


Not Very 
Confident 


Very Fairly Not Ve^ 
Confident Confident Confident 


Confident 


^ Fairly 
Corif Ident 


Not Very 
Confident 


, Coimaunicatlc^*'^ * 


63 ^ 


27 


0 


51 


\ 42' 


7 • 


77 


. 23 


0 


Cons clen t lousnes s 


62 


37 


1 . 


S4 


40 


6 


83 


17 








36 


0 


■50 


,43 


7 




'}'} 


' u 


J 

Independence and 
Initiative 


• 5.9- 


41 


0 


54' 


' 38 


8 




H 

9 ' 


.,0 ' 


ItLVolvement 


55 


44 

1 


1 


49 




9 


74 


. 22 




Persistence 


. 58 


41 


1 


^ . 53 


41 


6' 


74 ^ 

78 


17 


9 - 


Rasearclh ^ 


n 


36 


1. 


. 52 


.40 


8 - 


17 

« 


. f ■ 


Teaching 


56' 


38 




45 


40 


15 


55 


*1, 

4 


4 



The difficulty In obtaining cooperation from the planned nui^er pf 
departments was greater than anticipated, (In retrospect, there Is a sug- 
gestion that had the study been directed toward validation of the GR£» 
cooperation would have b$en somewhat better.) ^Faculty members In all of 
the departments' contacted Indicated some degree of being pressed for time* 
Clearly » many^elt that the research was of little direct value to them 
or to graduate education In their field. Department chairmen and graduate 
deans ^eemed more concerned with the problem addressed In this study than 
did the faculty. The difficulties encountered suggest tha^j^i^esjjread 
acceptance of scales of this type a^ criterion Instruments will not come 
easily 4 ^ f 

Thu3» the amount, and perhaps, to some extent, tH^'quality of the data 
collected for the pilot study were less than originally anticipated. How- 
ever, the faculty who did jcooperate used effectively the entire , range of 
the scales and felt at least fairly confident In the ratings that they 
made* Those Individuals who provided comments and sOggestlons, In addition 
to ratings, generally felt that one or more of the scales were not measuring 
a single construct. In the expected behaviors provided. they saw evidence of 
two constructs. This Is clearly a problemt'and suggests that some of the 
scales need additional work, 

, . The obtained reliabilities of the scales for one rater ^re. In most 
cases, not acceptable. This, In and oj^ Itself, does not clearly mitigate 
against use of the scales. The reliability for two raters Is reasonable 
for many of .the scales; In sltuatloVis wherff- three to five ratings (perhaps 
student and faculty) could be obtained for each student, th€ reliability of 
the average rating would probably be more than adequate* 

The correlations among the scales are quite high. This suggests that 
a large halo effect Is operating or that the scales are measuring less than 
eight separate construcfts. The patterns of correlations with other variablea 
provide 11-ttle help In this regard. If one takes thfe statistical signifi- 
cance testing seriously, few correlations can be considered as other than 
z^To* The. correlation of the scales with average pijellm scores In Psychology 
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Department .A (probably the best alternative criterion data obtained) supports 
the scales as being valid but the uniformity of the level of correlation also 
supports the Interpretation of the correlations among scales as a halo prob* 
l6m* ' , . - / 

Clearly* mote work needs to be done on the scales before they can be 
used in an operational context. Under the right set of circumstances* how- 
ever, it is felt that scales such as these could be used. One of the most 
"direct methods of increasing the reliability of ^ratings is to Increase the 
Quzaber of raters. In light of the results of this study it does not seem 
feasible to collect three or four fscidty ratings* but it might be possible 
to collect, say» two faculty and two peer ratings. When one considers that 
the usual yearly grade--point average is based on a total of st least 8 or 10 
separate ''ratings" of performance it does not seem unreasonable to require 
at least 3 or 4 ratings per student. 

Another approach which might prove more profitable is to involved faculty, 
within each department in at least one stage of scale development.'' One 
strategy would be to present a department with the genersl definitions for 
the scales used in the present study and allow faculty to ajgsign th^ specific 
behavioral anchors » Faculty could generate their own behavioral exaciples or 
could select examples from a pool which would be provided. This approach 
would/have the advantage of adaptabildty to the particular characteristics 
depsrtment and might generate a greater degree of interest and com*- 
t on the psrt of faculty. * 

^ ^ One question which is undoubteoiy^ much interest to those involved 
with the GR£ Testing Program is whether GRE scores' can be used to predict 
ratings. Unfortunately* the data collected do not really proylde^an anewer 
to this question. The data which were available, however, certainly do no^t 
suggest ^y striking relationship between GRE scores and ratings. This is 
not unreasonable in view of the fsct that the rating scales were designed 
to be multidimensional and appear on the face to'measure factors other than 
usual sca^emic aptitude variables which are tapped by GRE scores 1 This 
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suggests that such rating scales would be more usefd^ in a research con- 
text vbere a variety, of predictors (e,g<, biographical data, interest 
8coi:es, etc*) were included than as crit^eria in GR& validity studies* 
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APPENDIX A ■ 
SUMMARY STATISTICS FOR BEHAVIOKAL EXAMPLES 



The sumaary statistics for all the behavioral examples included in^ 
the study are presented in the following tables (^1 through A-16^. The 
behavioral examples are grouped by the eight scales. The odd-nucibered 
tables present statistics for behavioral examples used as anchors in the 
final rating scales* The even^ntimbered tables present statistics for the 
remaining behavioral examples. i — ^ 

Each t^ble shows the ratio of Judges who assigned eacH behavioral 
example to that scale in the initial sort (Step 3) » an.d the litean value 
of. the example as assigned in Step 4. The distribution df scale valties 
is also given in each table* 

The frequency distributions of values assigned to the behavior^ 
examples in a few cases do not add up to 12. This reflects tl^e fact that, 
for reasons unknown* a Judge chose not to assign a value to the example 
even though it was part of the scale* 

In. the coltmm showing the ratios of Judges assigning behavioral 
examples to the final scale, one will notice that the denominators arf 
not always 12. A ratio of lO/ll for a particular behavioral example 
means that only 11 of the 12 Judges chose to ^sign that item to any 
scade at all, but of those Judges Who did assign the item to a scale, 
10 of th^ agreed that It belonged on the scale to which it was assigned. 
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Di«cribuefon of $cal« Values for Beh«vioraX ixaaplt* Jt^sed as Anchors 
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215 



202 
218 



I vould'expecc thl* ecudenc co: 

pr»cac A r«porc ac a cegional tonvcnclon whlth would be 
veil ceteived for Its huAor and scyl^. 

9ubaic vriccen vork thac would Incecestiag oiid 
croubla co read. 

\tsually be tfndersco^ when spe^ka, but fail co tSlc« 
poincs In writcen cotiaunication uridor&caodabl^. « 

prapara oral and vrlccen toasunitaclona aaeaingly uiuwar« 
thac cha backgrounds of hla llacenara or r&ad«rs nay not 
be cha sama as hia own. 

taka can paj^as co codpiece an asalgnad paper vhlch should 
ba toaplccad In cwo pagaa. 

praaaQC Idaaa a sai^nar^ p^per^. or taac la a poorly 
organised diaJoUited fashiod. . ^ 



Fraotiencv^ Diiicrlhiicirtn 







1 




2^ 


2,5 


3 


3^5 
^1 


S 




5 


11/12 






3.9 ^ 










2 








1 


11/U 






2.8 




1 




. I 


7 






V 




12/12 


t 




2,2 




I* 




I 


2 








* 


9/10 






1.9 


- ^ 




9 


1 : 












^ 10/12 " ' 




X 


1.* 


1 7 




S 














12/12 





lean fcoB ^'aricital lotidencs" acudy 

^ftatic of judgaa aialgnlng bahavloral axamplaa 
CO fiiul ac«la 



or 

J 



' Table A-2 

DL»trLb^tLon of Sale Valued for Mdltlonal Behavioral Exaapleo 
Assigned to Scalft» but NOT Used as Anchors 



ICATIOH 



ITtH 



Prcnucncy OlBtrlbutjon 
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I.VOttH^^rtct thi* itudeat tos 


X 


1 


1.5 


2 


2.5 


3 


3.5 




4:5 






- -alb ' 


;dU^l«]f ^ i<\u«uall]f accurata Add sensitive choice of wOrds 
In tfpadtlng and vrltlog* 


5,0 














3 


1 


t 


10/12 




nana^c a dliilcuic topic with considerable aklll vhen 
proaentlng s paper. 

present IJcds l!t forceful way» giving an Lmpresslon 
of well-thou^t-otit ond Indcpcnlk^t vl<rus* 












1 
'1 


1 
1 


4 
A 
7 


1 


^ i 


9/12 
lir/ii 
10/11 


221 


artlcuUtely defend his position and l^^as. 4U 
aaopt ana ftamtam a wLCty artq urbana tone when appToprlatfl^ 


A;4 
A>1 










2 


1 


5 


1 




11/12 


aiz 


deal affectively wltt} tha denotlva and connotatlve valued 


A 1 

H 1 1 










S 


1 


1 






11/12, 


. 224 


present written and ^orel reports which would be orQaDlsed ^ 

and have clear introductions and lo^tal conclnslonuC ■ 












3 




6 






11/11 




deal effectively with the lexicon end grapsar of the 
Iajtguage\ 












2 ■ 


1 


6 




* 


12/12 


203 


Oreo ere revorte i/hlch are cl<Ar and well— oraxnlzed ^ oral 
or written ^ 


4.0 
















T 


12 


11/12 




> 

Ti«i -1 itr 0ATt * sai n E uHq re his qaqa rfi can An 1 V bc read 

by a specialist in his particular arcs. 

euboit papers vhlch arc extresicly verbose^ . . 


1.9 




1 


s 

9 


1 


A- 
1 










12/12 
9/11 


£16 


use a phrase such as "Vou^Knov'\ In no less than every* 
other senten^ when epeaklns In class* 


1 »0 






s 




1 










u/11 


204 


present material in a disorganized fashion so that an undue 
amotmt of tioe le needed to understand the points b^ln^aade* 








7 














a/ 12 


ctOl 


be unable to state in en unaebiguous way the key definitions 
end vuin points covered by the s^ubject Gutter. 


1.4 




1 ' 


4 














9/ii 


219 ' 


displey an^ inability to write competently* 


1.-3 






% 














9/11 


20s 


t&lk freely but eay nothing constructive; *to ude big 

words, often erroneously; to be almost laposslble to follov^ ' 


1*1 


11 




1 














11/12 


217 . 


subtait e report which Is replete with gracnatlcal errors. 


1.0' 


11 


1 
















10/12 




subnit work which ^ full of isistakea, incomplete sentences 
and misspellings; alaost impoaslble to follow the line 
of thoiiRht. 


1.0 


1 ni 




i 














10/10 



Htdo of Judges esslgnlng behAvlor&l exas^les 
^ ^ttel ecAle 



TabU A-3 

E>l«cributioit of S^le Vslu«« for BehAviorsl Exanples fJsed att Arithozs 





L7EK 




' X 


I 


l.S 


2 






0.5 


4 


4.5 




1 - 






I would expect thL« acudent to: 




























S02' 


r«tura with a carefully anootat^ bibliography after havlaj 
bein asiigned to tracV <Iovt) ^^^uiaber of referesceSi 




4.6 














3 . 




9 




5/12 




h^va the, reputacion anong^Jr both Che faculty and h'ia 
- fellow fttudeotff for dolir^vhat ha says he vill io* 




4.4 














2 


4 




11/12 


eoi . 

806 


■tldoB sIji daadlineSi ^ 

do esch assigned literature seorcl) b^at._each vj^^tiif be 

. ^ 7^^- ^ 


• 


3.7 

a. 3 






6 


1 


\ 
3 








1 




U/II 

9/12 


605 


flil on one t>r oore occaaiona .to co=tpUce a ttajor 
^aasignoent on tioe. 


X ^ 


l.^ 


a 




7 


I 


. 2 












9/12 


di2 


exhibit carTleaanesa with laboratory equipments 


X 


1.6 , 


s 




7 
















10/12 


B03 


«iaa elaaa repeatedly without contacting tbe' instructor^ 




i.a 


10 




2 
















* 9/10 



^tcL iroa "critical Incidantc** atudy 
2 * 

Ratio of Judges afiiignlA^^^>«t^av^raI exaoples to 
fitul acala - * ^ ^ 



J3 




Table / . , 

Distribution of Sc He- Values for Additional Hchaylord/ Exiepletf 
Asslgned^to Scales "bdt HflT Uaed as Anchors/ 



fctl«t 



807 



804 



C0HSCIEgao»Sy gS3 

^ ■ , ITEK 

I would exj^act thia student tat ' . 

huadla even the MSt ncrrlal asalgmnent {e^g,*, p«aper grading) 
Vlth care and renponalbllity^ " - 

have, i^reputation ialling'£o app^^r for one or tw^% 
profed$i6naI appointjaents eacfi year though he cotces on time 
to most ftach appolnta<int«'': ^ 

submit s report whl^h Is incomplete. 

fall to 40 backgr^lkr^ding^for s research project* 

^sg^in ta*Ioak f or^^ appr{>pri^te refer^aces a d^y before 
hs Is to report tt a seialnsr. 

1st sssignments slide .then, either to suboilt a iiastlly 
prepared report or to' sabait the r^pgrt well past the 
. deadline* ^ ^ 0^ 

fsU to appear lor a syckposium in which he was t^^^ a 
jpardlclpaot vlthout warning the other particlpaats* 



Hutio Judges atslgnlng behtvlorsl exaaples 
to llnsi scel* ' „ 



4.5 

1,5 
1.5 

1.4 
1.0 



7 
12 



T 



i frcquycy i)i<*trlbtttdon 



1.5 



2.5 



3-5 



10/ n 

10/10 
9/11 
-10/ U 

11/ U 



iiyii ' 



9/11 



ERIC 



40 



Pis trilmri^an of Scale Valws for Behavioral Exitfttfilee Vstd t8 Ahchors 





I would expect this scudent to; ^ « 


Cl^ 1 


X 


I 


1,5 




2.5, 


*3 


3*5 


4 


4,5 


5 




' 714 


ft pctceptive analysis and evaluation of a difficult text 
hie ff^ld of spccializaclon. 


I 


4.8 










'l 




1 




10 


10/12 


706 


&Ffor well founded quallficacions co Instructor's statetcencs 
In class Ih a discerning^ constructive way. 




4.5 














5 


1 


6 


iO/12 


711 


|sk qitescions which are always televanc and usually 
^srceptivei 


X 


4.0 




m 






' I 


V 






1 


I* 

10/11 M 


710 > 


be .sensitive to faculcy svaluacions and suggestion^ ha 
could incorpotate suggestions once his deficleoCrtes^were 
polnc^d out. 




3,8 








4, 


7 




1 


'! 10/12 


709 


auccsssfully idencify a ptoblea with another student's 
re^eatch but to be extreoely harsh In his criticissi 




2,9 






4' 


J 


7 




I 






- 9/10 




often be unable to consider qew ide^sTbbJectively 
because of etrongly held prejudices* 


X 


1,5 


5 


2 


5 


. 












11/12 


701 


)>t. unreceptlve to new ideas and proposals even In 
situations where previous nechods had proven Inad^^te^ 




1.3 


9 




3 














. ' 11/12 



Ratio 



. 1 



Itev fro* "crltlceX Incidents" study 

^tio of Judges assigning behavioral exafiples to 
fifiel scale 



( 



43 



Disf^ibutioa of Scole V^luen for Additional BehAVtoral Exiople 
' Aa^lgned tQ Scflles but HOT 43sed as AflrhorB 




I ... 
7U 



705 ^ 

703 

720 

719 

70d 
707 



704 
717 
712 

702 



CtltlCAL FACaiTY 



> 



ITEH 



yge^uency Distribution 



Ratio 







X 




1 c 






1 


1 c 
J . pi 


A 

H 


4 . 2 


c 






1 would expect this atudonc to: . 




























cotifll8t«Atl7 offer well founded and co^fttructlvo criticise 
of other student!* pr^entatioftB. 




4,7 








V 






3 


1 






11/12 


ft3(Ui^e carefully all authors* premises and* frane^ of 
reference before accepting conclufllons^ 




4.^ 














4 


1 


7 




9/12 


display 2n openness to evaluatlOD and crlticlen of his 
work by others^ 






y 








1 




4 




7 




10/11 


dstecc Inconsistencies la the position taicen by a 
professor in a critlcalt significant classrooa lecture^ 




/.I 










2 




5 


I 


3 




9/U 


bsck s position objectively aiid without defensiveness in 
s discussion on the oerit of sa Issue. 


4.1 










I 




6 


1 


3 




9/U 


s4opc a critical position prosper and appropriate to the 
work un^e^discussion. 

be able to understand ahe tone of a work and its 
underlying assua&tipns. 




4,0 
4,0 










3 ■ 
3 


1 


3 
5 


2 
2 


' 3 
2 




10/12 
9/12 


generate snd adequ^ely support a critical gene rail zst Ion, 




4.0 










4 




4 




4 




■ 9/12 


display ability to shift his point of view during a 
debate or discussion 




3,9 








\ 


5 




2 




4 




9/12 


sssess the li»itations of theories and principles froo a 
discipline when they are applied to a specific situationt 




3,d 






1 




i 5 




2 




' 4 


1 

k 


10/12 


be v^'ty facile and articulste, but his interpretations 
would be alvosc always lifappropriate. 




1,6 


5. 




7 
















d/11 


ftske exaggerated clalBS for the relevance or importance 
of his particular speciallcy and be unable or unwilling 
to deal with alternative approachest 




1,4" 






3 


t 














9/10 


oaks unwarranted assuoptioos b«sed on erroneous 
inforuationt 4* 




1,3 


10 




















10/12 



Hlatio of Judges assigning bshsvloraX exa^laa 
to fln«l scale 



44 



Table A-7 

' Dlacrlbutlon of Scale Vrtlucg for Rchovloral ExjiitplM Ugcd « Anchort 



ttfDSPESCDENCB 4 tlftTtATtVE 



Frequency Dl^crlbuclon 



Sutlo of iudg^ u^tlgnlng bAh«vlor&l ticAspIes ca 



R:itlo 



■ V 


I would «xpccc'%hlft ecuJont coi 






1 




2 * 


2,S 


3 




4 


4.S 


s 




312 


learn tn l&portont rescACCh on his own. 


X 


4.8 














2 


2 


8c 
6 


9/11 


31S 

1 


display M ability to fotmilace proble&a or Ifsuu suggested 
by oacerijftl vider acu^y rachec than the oedlaclon of 
bltf pcofcssotft,. 


















S 


I 


302 
301 


ask quesclons sad aeek' lufom^tlon beyond ^che tsaterlal 
In the cexc or lecture* ' 

develop « Use of several approprUc« copies for an assigned 
rtsearcb paper, but not co choose oo^ fron the list until 
rUtged to ^0 so by the professor afcec the deadline has 
pass'ed. 

depend upon his collahoracor for the suggestion of a 
toplCf deflnltl^ of the probleo^ and direction of the 
vork in a jointil^roject* 




3.8 

2.2 
2.0 


t 

I 
2 




r 


2 
I 


3- 
2 




>8 




I 


9/X2 

9/10 
10/12 


314 


not respond to euggeaclons or supervision unless the / 
Instructor pursues^ have' to be prodded and pushed and 
pullsd along. 




a. 2 


.10 





















47 



Table A-8 

DiBCrlbuti<»i of Scale Volues for AddlClonol Behavioral Eit«opl«^ 
Aafttcnfd to Sealca but NOT Uafrd Af^chora- 



( 



ITEM 



Frequency &lgt^ibtitlon 







3C 


1 


1.5 


2 


2.5 


3 


3.5 


4 


4.5 


5 






1 would expect this «CudenC Co: 






















307 


Cflke on challenRing or "no^ ycc officially ;jpprov<>d" 
problmt or iirrtuo«« 






i 










4 


I 


7 




304 


do IftdepdndenC t«d4ing or research Co check the validity 
of InCfttpreCaCions or evaluaCiom which differ froo Choae 
of hit iiidCrucCor or claasoaCes. 












1 




I 


1 


9 


9/12 ^ 


313 


becottc raarc proficienc In a t^efiil outside fiel<t under 

11* V VWU all 4 V V a 4 














i 




2 


6 


U/l2 1- 


3» ' 


obCain a eopy of. an unpublished research reporc or ^ 
paper relevanc Co hia ovn work Chrough correspondence * 


4.3 










4 








7 


" 5/12 




seek lurcher infprmaCion iiidependent of clas^ ^sslginiienca, 
when a professor raiaed a qu^scion in claas,. in an 
aCCettpC*Co anaver che quescion more e^iCensivelXr ) * 


4.3 












, 1 




1 


3 


' 9/12 . 


. 308 


regularly spend Cime looking chraugh che approprlaCe 
journals when cheae are noC pares of assignaenCs^ 


4,2 










2 






1 


4 


8/12 

I*- 


316 jf 


fanillariza lUmself with Che resources of neighboring 
libraries and special colleccions^ 


4.0 










'3 








3 


r 

8/12 


305 


nc^ an insctuccor'a help in finding a Copic when 
required co prepare an aaaignaenc on a self^^el^ced. ^ 
copic. 


1.0 . 






7 


1 ^ 


2 










' 10/11 


311 


, conacancly seek help frOD faculty on crlvial aacc^rs. 


1.0 


k 


1 
















10/U 


310 


be heavily dependenC on d^recClon froo facuXcy ^d 

appear unable Co undarcakfi indepeadenC investlgikcionaf 

1 


1.0 




1 
















11/12 



SlAClo of judgia aaaigoing behavioral, cxamplea 
CO final acale 



48 



4[) 



T«ble A-9 

PUcrtbudmt of Scalfl VaiWa for Behiivlor^l Kx^Plet Uaed >a Anchora 



• 


ITEM 




J 4 


1 ' 


1.5 


2 


2.5 


3 


3.5 


4 


4.5 


5 






I 9mild txpecc thla student to; » 


























603 


be elected u so oEfli^r lo « reglonsl or loc«l 
^ {^roftttftflonal society.^ 
















" i 


2 




9 


12/12 


617 


becooe quickly and ettthuslBstlcally lavolved In a projects 


X 


4.5 














6 




6 


10/11 


605 


display concern and tnter<s3t in work being conducted by 
faculty. 




4.0 










2 


1 


7 




2 


8/12 


415 


^ t 

attend deparcaental seialnars but to neither prarclclpate 
In the discussions nor volunteer to present a report. 




2.7 
1.8 


t. _ _ 




3 


, 1 


a 










a,0/12 


611 


not attend a laeeclng, either local or natlon^l^ of tbe 
appropriate profesalonal aoclecy. 




1 


* 
5 




2 










11/11 


^ 616 


avoid challenging courses or vork. 


X 


1.4- 


1 ' 


* 


5 














9/U 



Frequency Distribution 



Ratio 



^rte» frott "critical incldcnta" arudy 

^tlo of judgu a^clgnlng behavioral exaoplea to 
final .acale 

^Ttim lt«m vaa placed Incorrectly aa an anch4>r 
(at approximately 3rO) In the racing scale book- 
lat. Xta Bean Indlcataa chut It should have baen 
placad near tha top of thla acale (at 4.7). 



5 



Table A-10 * 

^ • 

Diitributlon of Scale Values for Addicional Hehaviordl Exasples 

A6sisne<i to Scales but NOT Used At^chorf_ 



601 
613 

601 

609 

69a 

604 

t07 

610 
655 

612 



tHVOLVEHQIT 



ItEH 

I vou Id expect this student tot 



give up 4 vacation to co-authot a paper with a faculty 
aetcborA. 

j;[ve on original pnper ot a convenC£on or pceting 
sponsored by a scholarly socioiy. 

dijipUy A genuine interest in and comLtaent to his 
field in infonaal discussions vith fatuity* 

wait (Qr the instructor outside the classrood after a 
particularly gooU Jccturc wiL^i an invitatlort to join 
it^ a furth>t discussion of the subject with a few otjicrs. 

appear to be very Xncerc^ted la oeeting coileagues^ as 
at coaventioas. 

hold Deobershlp.in the appropriate professional eocietjr 
or assodtdtion* 

be ^ student meisber of a national professional society 
but not an active participant r 

hold no oeobership in any professional society and to 
not, appear interested in doin^ so. 

attend se^^aars only when required to do $0. 

becoad distracted by non-acadeqlc» non-profeasional 
ihterests* 

sftldna^ if ever» engage In infonaal contacts- vith 
_facultj or fellow graduate sn^dents. 



5.0 

4.4 
4.0 
3.5 



1.6 
1.4 

1,1 



rrcqugftCV t)iittrhi:ttiinn 



4 
7 

7 

10 



1.5,, 



2.5 



I 
1 

2 
6 
10 



3.5 



^ 4.5 



^ao of judgsa assigning behavioral exanplsa 
to final scale 



52 

ERIC 



T&ble A-11 















t 

Ftoquency 


Distribution 


; 










X 


1 




2 ' 








■ 


4.5 


s 






X vouU expect student tot ^ ,^ * 


























' ' 508 . 


purAu« hla Interest or ideas despite dlscoura^ng ftxtvlce 
frofii feculty 4fut to b4 succeesfui^^ ^ ' . 


X 


4.8 ^ 










• 




2 


1 




10/ « jfH 


■J 502 


try eveo harder whetr «i preblem oi supposedl/r »oder»t« 
difficulty roelete All Initial attempts to resolve It. 




4,6 














5 




- 7 


, 12/12 


: 514 


repeatedly esk qoe$tion« of faculty until he fully 
undctfltarvd« «n lasuai ' " 


X ■ 


4.1 




> 






2 




7 




3 


a/12 


509 


frequently talk about leaving school ad sooi( as a 
' .wtet's pro^roa haa been cotnpleted chough ti^^ 
entered graduate school to get a ilpft;ora(e. 






. 1 


1 




1 












• 

6/10 


* 513!,' ' 


abend on e project after losing a aet t^f preUelnaty ' 
* date. ' * ^ ^ 


" f 


1.3 


8 J 


. 1 
















11/12 . 






m 
















i 


i. 





t 



Httm froB '^G^ltlcol lacldente*' etudy"^ 



TKiit:13& of j ^ateignin g~^ eEevlo roT'ex^lee to 
final ecale 




r- 



55 



^ DUcribucion of ScmU Values for B«tutv£oraI Ex^npUs ^ 




403 
AW 
404 
-401 

403 

407 

411 



XT5K 

I votild cxpecc th£4 ffcudenc to\ ^ 

dtv«Iop «n origimil way of hflndling « rea^arch problea. 

b« f»£I£4r, vith tlie lacesc developaencs in his field* 

b« 4ysceti4t£c in hi« 'g^chering «nd order£ng of ddcs. 

repIUoc<3 previous research done by^cb^rs tfich ^ 
CAr<3£uIIy*cofiducced and vftll*r6pbrced expericetvc^ 

cpTVducc s fa£rly rducine and un«xc£c£ng research . 
projepc* 

confine liii acteittioa co research naccera of sisor 
b« tiaable to effa^kvely apply a parcicular research 
acccftpc CO carr7 <nic poorly plaimad reaearch^ 



CI 



4-5 

2 
I 



U/12 

9m 

10/12 , 

8/11 

9/12 

.12A2 
10/12' ^ 



Ices^frott ^cricieal incidcncs" acudy 

^^Kitio ot Judfti msaignlng bali«vloral ascanplev to 
£Jiul^4cal« 



^ Divcribucion of &c4l« Valuct for Behavioral Exonpl^e Uft^^ 



BESEAJtCB 



Frequency DlgcribUclon 









X 


t r 








3 


3.5 


4 


4.5 


5 






I wouX4 «xpecc chit tcudenc co: 




























dtvftlop tn original vay of (moling a re«^arch problen* 




4.8 














2 


2 




- * U/12 




bft f tall lAr« with the Ucesc ^evelopo^ta in hi5 field* 


x' 


4.3 












1^ 


7 


I 


3 


'8/12 




bi tysceoaclc In his 'gftcherlng and ordering of daca* 














2 








1 


9/U 


'401 


repllcikcc previous research done by^chars vich a 
carcfully-^^occad and vell-rcpbrtad cxperitjent. 




^.5 ^ 










5 


1 






r : 


10/12 , 




conduce i fairly routine and unexciting research . 
project. 














U 








mi 


^ 407 ^ 


confine' hia attention to reae^rch ciattaxra of t&inor 




























isporcancft* 




2.3 ' 


■1 




6 


1 


4 










9/12 


411 


be unable to tffe^^ely epply a particular research 
techniquet - ^ ^ 




1,6 


4 


' 2 ' 








*■ 








I2A2 


409 


etceapt to carry out poorly planned research. 


X 


1.3 


a 


2 


1 


1 




*■ 


1 






10/12" ' 



^tes'froa "critical inci^cnte'* atody 

^^tL^ of Jud{M eeaignlng behavioral exaasples to 
final acale 



Tahlt A*1A 

Distribution of Sc9i^ Valuei for Additional Behavioral Exasplta 
Assigned CO Scalea but ifOT Used aa Aachora 



ScaUi MggAnar 



F ro Quanc y D tj tr j biit Ion 







X 


1 


la5 


2 


2a5 


3 


3-5 V 

a 


Ik 


4a5 


S 




a 


-I voold 6Xp«ct thia atudenc- toi 


^a 












1 












WMZMT a difficult research talthni^ue in an unusually 


























abort perldd cf ttna. 
















3 


2 


7 


10/11 




d«al insightfully with primary aourcesa 


4.3 


'1 

\ 








a 




S 


I 


7 


! 8/11 




Tftly too haavily oa onCa reaea^tch cool \n coaductlag 






















a 


Tt^aarcha 


2aS 


i ■• 


2 


4 




5 










11/12 




b« tmabla to fomulata a uatable hypocheaia froa a 
























Cbaoretical analyaiaa 




i ' 


' 1 


6 




1 










9/11 




conduct raaaatch without providing proper controls 


la4 














/ 






zm 




ao that rasults ara queatioo^bla* 




S 


3 








• 






. 4ii 


b» unfamiliar vlth a &aJor ta^earch . tool in hia f laid a 


la4 






S 














10/11 



^Sdtio of Judgaa asaiffs^ bahavloral axaaplaa 
to fliul acala ^ ' 



. J 



,60 



IHscrlbutlon of Scale Vaiuei for Rct^iyloral £atattpl« U>ed Anchors 



l!)3 

lOS 

119 
109 
US 

ICS 



1 vould lids stiif^^ Co; 

•bov laA^;]^tlon jond brlglnail^ in teaching a tr«dltl«i4ll7 
top^ to «n uodergr^u^cd class- 

htlp slower stud«ttts VDlu&tarlly. . 

*coftil^erabl« pains t& help undft^gradu^tes with thalt 
Xv^rk' ivaa J^j^gh his pres^Caclon of suterial in class 
Is frA^^ntly poor. 

ipa&d class tine doi^g routine exercises^ 

coaplaiA obouc having to teach the introductory course* 

teach a class in vhlch hia students seek help froa other 
Instnictori. 

bodgsr students ^M be ^erslly unsynpathetlc co 
legitimate request for cIm extensions or specific help« 



Frei^ueticy PUtrlbutlon 



CI^ 1 




1 ; 


1.5 


2 


2.5 


3 


3,5 




4.5 


5 


X 


4.3 










. 

1 




1 




U 




3-0 
2.A 
2.0 

1.6 


2 


1 
1 


1 
6 
7 

7 


I 


8 
3 
2 


2 






i 

] 




1.2 


10 




Z 















10/11 
^ 9/10 

10/12 
10/11 
9/10 

11/12 
12/12 



2 ^ 



Htem from** critical lAOldenciV study 

^Eatio of ju4gM osslgnlag behavioral escaipples 
to final scale 



5^ 



TabU A-16 



DUcribucloo of Sc«U Valuei for Addltloiul Behflvloral Ixompltt 
^ A««lgf>cd to Sc«lei but HOT Uft«d &« Aiicbori ' 



I would cxp«ct this ftCudent tot 



( 



Frequency Ij^ctrlbutlon 



X02 
U4 
U« 

116 

123 

104 
U7 

121 

112 . 
126 

120 

lU 

U3 

127., 
u6 • 

122 

124 

107 
O ■ 



jitiwldtB great Intfirest and «nthujslaa» lo un^ergr^duAtt 
courses 1ft tfh^ch ho 1« «n lftstru«tor. 

develop provoc«tlv« And/or lfiagUiitlv« o^rclsdd fdt , 

' utabliah Of) «tBofipher« In vhlch •tudents feel fre^ and 
ejigftr to calk. 

be •ennltlv^^p fAulty tvAluAtloiu and suggestlortd and 
ttf chot^Ro his teaching ttrotegico when deficiencies 
•ro pointed out. 

identify ett^ente vLth difficulties and make 
sppolntMnte vlth theft. 

^ revise and restructure an entire Incroductory^course. 

devotop crltori^ ^nd eet up c&nslfttent sCrmdards by vhlcb 
to evai't.itc jtuJent work. 

echediile office hours for his &tjder^cs ^nd try very Kard 
, to keep thet). 

pl^A^^and Ispleiaent ari effect syllabus for a course^ 

others for ideas on hov to present cooplex concepts 
. do his etudents. 

never miss a class he is to tesch but not ^o Announce 
office hours 

know the nattes of about a third of bis sciKiettts la the 
class of 23 students which he teaches. 

conduct highly etructured classee oarked by a lack of 
flexibility end opeft Incerchs^ge of ideas. 

cover All the apprropriete subject a^tet In che course be 
teachee. 

eetabllsh rigid- criteria and a eec of Inflexible standards 
by which to svalu^te student work* 

OODS to classes he teaches but have no lesson plan* 

bs sntreaely eeneltlve to student pressurs or crlclciaci 
vhlch would be reflected In the ssslgnoent of poor grades. 

teach a clasa vhlch ths bstcer undergraduate students 
consistently cut* 

tasch s class in which s largs percentages of tbe stfkdents 
drop che courts to switch co anothsr ssction. 

curvyr fsvor with his students^ ftoka thajdensods oi^ths 
dloci^Jioe sscondary to a dssira for popularity. ^ ' 



5.0 
4.8 

4.3 

£.2 
4.2 

3.8 
3.6 



2.8 
2,3 
2.2 

2.0 

1,9 
1,7 

1.6 

1,5 

1,5 

1.3 



'2 
1 



1.5 



2.5 



3.5 



4^5 



12 ^ 
10 



1 

^2 



. Ratio 

--lWl2 
12/12 
12/12 

10/10 

12/12^ 
10/11 

12/12 

> 9A2 
11/12 

9/11 

9/11 
9/11 
10/10 

10/ U 

10/11 
lO/n 

' 9/10 

12/12 

Xl/12 

9/11 



^Llo of judges osslgnlag behavioral s^caaplas 
to final seola 
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APPENDXX B 



Criterion Hatijig Scale Study 

INSmUCTlONS FOR DISmiBUTING AND CODING RATING SCALE BOOKLETS 
AND RECORDING INFORMATION ON STUDENT RECORD SUMMARY FORMS 



The peraion In charge of gathering Information should carry out 
the following activities: 

• Determine Which raters will be evaluating which students 
(an attempt should be made to see that am lndlv^.duai 
rater is not asked to rate an extremely large ninnber of 
dtudents). * / 

. , / 

It See that each professor receives hj.s rating ^slgmcents on 
time. ' 

r 

• Sollow-up noti^-reepotises at the appropriate/time. 

"/ ■ 

• Arrange for rosterJng reat scores and ot^fer Information 
Requested from studtmts' files. , / 

• See that all information is organized jknd mailed to ETS 
as soon as possible. ^ 

* y I ' 

i 

Rating Scale Booklets ' / 



\. T^fkeep Information confidential^' is Suggested that the 



data coordinator assign an arbitrary ni|ftibe^||to each student. A number 
should also be assigned to each faculffy rater. The coordinator should 
keep a master list of student^ and fealty Identification, nuc^eifs. When 
the ratings have been completed^ t^e coordinator should wxlte in the 
identification nunibers ftom the iftaster list on the eovers of tljie booklet 
and strike out the students* and raters* names. The coordinator should 
retain the master list untl]/ETS has received and edited tl^data.^ At 
the completion of the study ETS will destroy the i.fidividual data and 
thfe master list should be destroyed. 



^ 2. It l8 essential that every, student be rated by his^-o^r her 
faculty advisor. In addition, ve hope to obtain ratings by a 8ecand[ 

professor who Is familiar With the student^s work. It'ls not necessary 

- - ' ' ' ' ^ \ 

,,tfaat all students be rated by a second pro|^&so^ but It Is essential 
that we have, two ratings for at least two-thltds" of the students being 

,^a£ed. The **flr^t rater** sKbuld be the student *s faculty advisor. * 

" * f- * 

Sis ratings are ^to be Identified by the number "1** to be written In 
the lower right comer of the cover page of the rating scale booldets^ 
The '^second rater" Is to be Identified by the number ^*2" In the lower 
right comer of the cover page of the ratli^ scale booklets. 

3'i It will not" be necessary for the data CQor41^tor to record 
any of the obtained ratings. ETS will dq-thls. 

Student Record Stmnaiy Foms 

1. student (Ho,) — To keep student Information coi^ldential » 
stxtdents will be Identified oi^ly by- number. The number^ assigned to 
a student for the rating scale booklets should also be U6*ed here. 

N ) — The ^ 

number assigned to him for the rating scale booklets. 

3* Second Faculty Rater (No.) — The second faculty rater^ls 
also Identified by the number^ assigned to^hljn for the rating scale 
booklets. € ' . 

4. Sex — self'-explanatory. ^ ^ 

5. Number of Semesters E5iifolled In Graduate School — Recorlfthe 
total, pumber of semesters enrolled. Including Spring semester, 197Si 

6. Area of Specialization ~ If student Is specializing In 

more than one area of the field, please 'list all. ^ 
% \ - ' 

- ■ _ 



2, Advisor (No,) — The faculty advisor Is Identified by ;t 



In GRE Scores — Self-explsnatjyfy. - ' * ■ 

^ • 8. Iftidergr^duate 6PA — T^elf-fexplatfatory. 

9^ ^ Graduate GPA — Putnlsh" the most current overall GPA for the 

'student* Xf grading system ^6 other thsn A.O = A, please advise us. 

lO. Core Courses — List course titles and grsdes. > 

.11. ' Additional Information ~ If any other Informatlfan Is svallable 

(for'exantple, tests *^gtven as evaluative Instrupente* rontine evaluations 

conducted during graduate work, Van^rderlng of^ students)^ wpaUr*"' 

f ' ' I 

like to have it. 



Send the compleC^d tstlnfe boosts snd student, reqord forms to^ 



. ' . i)r. Richard R.'Reillwf 

Research Psychalog;lg»^ 
Developmental ReseaBlf^Division 

' ^ R-226 ' 

\ - Educational Testing Service 

- * Princeton, NJj 08540 

Notify £TS when return ^shipment has tteeii made. ' - 

' If there are any questions or problems^ please call Mrs. Peggy Itehoney 

at (609) 921-9000, Extension^'2383. *t * ^ .. 



■ ■ -r. 
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Oa Ehe following pAg^fl '«lghE facEors which hav« been IdeaElflefd 
A» l^otE^E In the p^rfomance of {tt^^aEe studenEs are described hy 
«aaii« of a' general definition. In addition, various points along the. 
<luallEaElv« scale ara "anchored" with specific behavioral exaoplaa, or 
inddanEa of graduate sEudenE perfomance. For each attidenE yoxx rata' 
placa a check natk aE;Et^e polnE on the gcale which' In your Judgment ' 
baa^ daacrlbaa Eha eEudant'a performance on EhaE parElcular EraiE. AXao^ 
i^lcatJi for each Erait Ehe level of confidence you have in your rating* > 



StudanE being raEed_ 



Rated by 



RIC 



r 



CommmlceClOD 



General Definition 



/The ability to tranffsit ideas and feelinga^ It includes the degree 
of organisation and precision, it involves the extent of Undetatending and 
perception and inclutTes both verbel and nonverbal transmlj^siona* 



I-WJULD EXPECT THIS STUDEOT TO; \ - 



preaeot,^ report at a regional co:^ventio:^ vhich would be veil received 
for itb tiuvor and style. * 



subttlt written vork chat would be interesting and no trouble to reed* 



tift^lly be uoderstci^ when he speaks, but fail to make points in vrittan 
poBDxmicetion underat^dable . 



prepare oral e:^d written conaunications seemingly unaware thet the Backgrounds 
of hia liste:^ers or readers may not be t^e same as his own. 



take ten pages to coiopleta en assigned paper which s hould be cottpletad in 
two pages. , , * 



\h should 



preaent ideits i:^ a s^nar; paper^ or test in a poorly organiv^ and 
disjointed fashion. 



t ^tf ' ^ Degree of Confidence 

□ Very Confident ITI Fairly Confident " I | Hot Very 

'I — I ' * — ! Confident 
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Cons cl «at 1 oufl na« ■ 



fianiTAl Definition 



ataiidSda, Includes extent of carrying through ou tomit^ts eveur wtieo ^ 
they are w>f folly aj^led out. * , 



I WOULD EXPECT THIS STUDENT TOx 

i. 

«tum with . ««£«lly ««»t*t.d -bibliography after h-vlng been «.lgn*d 
to tr«k down » nustoer of refaMMed. 

Uv. tto «put«lon «on8.t both th. faulty and. hi. f<dlcw »tud«» 

for 4ola8 *h*t l» a«y» h6 wUl do. . ^ 

■ . ' '] ■ 



aeldoa ttiaa deadllii&s'. 



do .«h aligned ll«r«ur« .«reh but e«h would be iwot^l**- 
fai on one or oor, oc«.lon. to co«,U» » «i« «.lgn«nt on tl«. 

Mthlblt wrelM«w.B« with labor»tosy '«l«iP»«»'' ■ ^ 
. mtis CUM tipe-tedly ighouf eontsctlng 



Very 

'confident 




^ ^ Dasr^^ cotifldefiCB ^ 

Q very ConfXd«.t . F.lxiy Confident ^.D^JL 



Critical Facility 



Gener al Definition 

— \ . f 

The ability to evaluate the products of others and offer alternative 
t^ypotheaea^ isethoda or analyses when appropriate* locludes the ability \ - 
to y>i parausdai by veil reasoned arguments even when they are tlearly ^ 
critical of hla own vork or position on an iesue^ The ability te identify 
^probXaas and structure priorities/ 



/ 



r ifeuU) EXPECT THIS STUDEHT TO: 



aakc a percdptlve analysis and (evaluation o% a difficult t^xt In hl^ 
flisld of specialisation' 

offer veil founded quallflcstlona to Instructor's state«enta In class 
In a dlacernlngr constructive vay^* 



ask questions which Are always relevant and usually perceptive^ 

/ 

be sensitive ^to faculty .avaluatlona and aaggastlona ; he could Incorporate 
suggestions once his deficiencies. ware pointed out* 

J 

successfully identify a problem with another student^s research but to be 
axtraoely hareh In his criticism. 



of\en be unabls to consider nev Ideas objectively because of strxjhgly 
haldy prejudices, ' M 

be uiu^aceptlve to vam Idjeas and proposals even In situations vtiera previous 
MthodXhad proven Inade^iiaXe* * ^ ' ^ 



Confident 



Degree of Confluence 
j Paijly Confident 



Rot Vsry 
Confident 



t 



■ f 
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Indepeodence snd Initiative 
' • Cftnftral Definition 

Ih« coabinfttioo of self confidence and drlv«. It includes taking re* . 
iponsibiUty tnd tthovtng ittitlAtive. It oay reflect, intellectuel curloaity 
tnd BQtivMl^. ' . , ^ 

/ _ 

I U(nJLD KXFBCT THIS STUDSIfr tOi 



lefttn IS iaportent resesrch ekill on 1>1« ovn, • 

dlfpl«7 tn tbiXity to fotnulete problafis or iiauet euggeated b7 the utetltl 
itndet itudy ratberr cbm tbe aedlAtion ^f hli profMSors. 



ok queicione and seek infotrutioo beyond the naterlAJ. in the text or lecture. 



develop s lilt of teverel epproprlAce copies for An designed retsArch peper 
but not to choose one froa the liet unCil urged to do ao by the profeeso^ 
^cer cbe deedlins hss peesed. 

defied tspoti his c^llaborscor for Che suggestion of a topic, definltioo of the 
problem, sad directioo of the vork la a joinc project. 

n^ respond to suggescions or supervlsioo unXesa the instructor pursues; 
bsTs to be prodded end p^hed along. 



Degree of Confidence ^ 

i 

Very Confident Q Fairly Confident Q S^^^ti' 



i \j 



Involveaent 



"-^ ^ Caneral DeflnlElcm 

The deg«e o£ participation atwJ activity In botti and iaf oraal 

Mfltaxts related to the dlaclp^lne. The extent to which ^tereat and 
SS"la« laTxhlblted and 4l«talned. Could be Interpreted ea the extent 
- - to vhlch a coBttitbent to the/field has been aade. 



I WOUU) roSCT THIS STUDEHT TO: 



becow quickly and cnthualaatlcaUy Involved In a project, 
dlaplay concern and Interaat In work, being conducted by (acuity. 



be elected ia an officer In a regional or local aoolety. 

attend dajiarowntal ae^nara but to neither participate In the dla«iu.Blon» 

mr volunteer to prwicot « report t 



nafi^'«tt&d & ««tlng* tither lo^(l or wtlonal. of the appropriate 
proftMioiul ftoclAt?* 

tvold Ghallenglug <ioi*r«eft of work. 



Degree of Confidence . 
p very Confld«>t Q Falriy Confident Q 
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FerslBtence 



Ggneral Definition * ' v 

The characteristic of contlnttlng to ptirstte a tadk or Idea despite 
criticise or ftet^Acka. 



I WOULD mSCT THIS STUDEHT TOt 



pxjrsus hla l^tere'st or l^ieaa despite <ildcottra;;lng advice from faculty and 
to be attecesaful . 

try even harder vhen a problfnjoC aup{>^aedly uoderete dlfflcxjlty reslstt all 
Initial atteopts to resolve it! 

repeacedXy aajt questions of faculty until he fully xjnderatanda an lasuet 



frequently tellt about leaving school aa soon aa a aaacer'a progr4» ha» 
been coopleted thot^h he entered graduate school to get a doctorate* 



abandon a project after losing a set of preliJiiinaTy 4a.ta« 

Degree df Confidence 

□ Vary Confldenr I I Fa^y Confident" | I Wot Very - 

I — ^ I — ^ Confidtnt 



ReseaCch 



General DeflalCloo^ 

Concenied vlch curiosity ccupled with Che desire tio expBud or refine 
knowledge Ln the^ discipline* The ability to Identify essential elesents 
of coe^lex pro&lens end to fonsulate and sijpport'a critical s^erali£«tlo£t 
or hypothesis* Breadth and depth of knowledge of nethode and procedures 
approptlace to resea^kxoupled with the ability to use the toola noat 
relevant to the Investigation to be undertaken. The ability to plan 
research adequately and to-carry'lt to coaplatlon with care end precision^ 
yet remaining flexible and aenaltlve to the possibility that data cay 
Indicate a need to oodlfy preconceived hypothesea^ 

r WOULD EXPar this STUDEHT It); 



develop an original way of handling a research problem. 

be ftnillar vltb the latest dei^lopoenta in hia field, 
be ayateaatlc in his gathering and ordering of data. 

tepllcate pra^ou^ research done by others with a carefully-conducted and 
vell*taported eTparloent. 

bonduct a fairly routlna and imexcltlng reaearch project* 

f 

confine his attention to research patters of ninor Isportance^ 

be unable to effectively epply a particular reaearch Technique* 

attempt to carry out poorly planned tesearcb* 
0 ^ 

Degrth of Confidence 
♦ 1 

□ Very Confident ' I I Fairly Confidant | ! fhx Very 

* I — ' ' — ' Confidant 



7p 

* U 



Teeing 



General Paffnltlon 



Ih« ability to cocnunlcate concepts of atudextts, ability to iBtexest 
atudenta In aubject oatter^ Includes enthusiaaa for teaching* villingneai 
to 9fftnd clM both in praparatlon for claasea and /mt of claaa time with 
Btudanca. 



I mr[J> EXPECT THIS STUOKirr to: 



ahov Icagination and otiglnality in teaching a traditionally dull 
topic to an undergraduate claaa. * 



help fllover studenta voluntarily. 



take conaiderable paina to help undergraduates vlch their work even 
though his pnaantation of ttaterial in claaa is fre^&ently poor^ V 



apand claia tiiae doing routine exercises* 



covplain about having to teach the introductory course. 



taach a claas in which his atudents seek help froa othar inatructora^ 

badger itudenta and be ganarally unaynpatbatlc to lagitlntata ra^uaata 
for tiaa extanaiona <ft specific help* ' i 



* 



Degraa of Coaf idence 







